1. Field of the Invention
The invention relates generally to the field of image display methods and devices which display images representing virtual environments, and more particularly to improved methods and apparatus for producing the psychological effect of free and unlimited movement in a virtual environment in response to limited physical movement by a user.
2. Description of Relevant Art
The following description and analysis of the prior and present art is based on our working hypotheses concerning the emerging art of virtual reality. Since the body of knowledge associated with this new field is largely speculative, our hypotheses concerning the principles of operation of the present invention are not presented as a basis of its operation but rather an aid to the understanding of and full disclosure of the present invention. A virtual environment or world is one in which some portion is artificially simulated or controlled, often via computer. A computer may, for example, create a graphic simulation of a cityscape, complete with buildings, trees, people and so forth. The buildings of this virtual environment may be complete in much detail including furniture, lighting and even other people. Environmental sounds or other sensory inputs may also be so simulated. The generated environment may be viewed on a two-dimensional display such as a computer screen or by viewing with special stereoscopic equipment by which the scenes are made to appear three-dimensional. Other devices such as special earphones may enhance the experience of the human interacting with a virtual environment. The scope of the invention relates generally to virtual worlds or environments and is meant to include telepresence or any other environment in which visual information or other sensory input is manipulated or controlled. Telepresence herein refers to remote head-coupled camera systems of the type typified by the Molly.TM. teleoperated camera platform available from Fakespace, Inc., or other forms of supplying video information of this type such as from a prerecorded video disk.
The art is replete with invention relevant to computer input/output devices in the attempt to more effectively connect the human to the computer. The design of input devices and user-interface software has continually developed toward promoting more natural and intuitive human interaction with computers. For example, computer operating systems based on graphical desktop and file cabinet metaphors are often preferred to typed-line commands because users are familiar with look and use of desks and file cabinets. In the same way, the use of a mouse as an extension of the hand into the computer environment is a considerably more intuitive way to move things around in a computer environment than are typed control commands. However, while the mouse is successful as a two-dimensional desktop-metaphor device, it is considerably less successful in three-dimensional virtual environments. We believe there are special physiological and neuroanatomical concerns when connecting the human to three-dimensional virtual environments.
The salient problems of the physical human negotiating virtual environments have to do with the user's knowledge of and control of his position and orientation in the synthetic (virtual) environment. For example, in our everyday lives, when we move our head, eyes and other parts of our bodies in order to look at something, there are many and various inputs from the muscles, tendons, vestibular system and so forth which we have come to expect. It is this gestalt of the world which our brains create from this bundle of stimuli which gives us the basis for the control of our bodies in our everyday lives. Successful negotiation of the virtual environment then, is enhanced by the harnessing of these familiar but unconscious feedback systems which we humans have developed for control in the physical world, to the virtual world. Thus, the use of the hand alone, as in the mouse, to represent the motion of the non-hand parts of the body, for example the head and eyes, requires too much additional cognitive processing to engage the user in a fluid and intuitive connection to the virtual world. But imagine "looking around" by typed-line commands. All one-handed input devices such as mice, track balls, spaceballs, gloves and joysticks suffer this basic non-congruence of hand motions emulating non-hand body motions to control what one is looking at (or the non-congruence of finger motions emulating non-finger body motions to control what one is looking at). Examples of such devices include those described in U.S. Pat. Nos. 4,988,981 (issued Jan. 29, 1991) and 5,097,252 (issued Mar. 17, 1992) for controlling a computer in response to hand and finger motions, those described in U.S. Pat. No. 5,184,319 (issued Feb. 2, 1993) for controlling a computer in response to hand and finger motions (and providing force feedback to the fingers or hands wearing the devices), the Multipoint Z 3D Mouse devices available from Multipoint Technology Corporation, the Global 3D Controller device available from Global Devices, the Geometry Ball device available from CIS Graphics, Inc., and the Spaceball 2003 device available from Spaceball Technologies Incorporated.
The Spaceball 2003 device (as are mice in general) is formed for manipulation by one hand of a user. It provides soft control (by means of a key click button) for switching between a mode adapted for left handed use and another mode adapted for right hand use, and is not designed for interchangeably left and right handed manipulation.
Other conventional input devices include those resembling the yoke of a flight simulator or the steering wheel of a car, which control motion of a vehicle in a virtual environment. While more sites of sensory feedback are involved with use of these devices than with those noted above, the purpose of this art is control of a virtual-vehicle; not natural control of a virtual-body (which corresponds to the user's physical body). For example, with a virtual-vehicle, the physical-human's viewpoint is largely determined by the manipulation of the virtual-vehicle. Airplanes (or other vehicles) and humans do not have the same movement modalities. Thus, by way of example, a human's physical-rotation about his (her) vertebral column has no directly corresponding analog of movement by an input device of the "flight simulator" type. And, conversely, a push forward by the hands on the yoke of such a flight simulator will dive the virtual-airplane but a similar push of the hands in ordinary life will not cause the human to so dive. Even though the human interface device of a flight simulator is typically a yoke best manipulated by two hands, movement of the user's body does not generate corresponding movement of the airplane. Thus, in a flight simulator, to look left a user does not just turn his (her) head or twist his (her) trunk and look left but rather manipulates the flight controller yoke to bank the virtual-plane left. In summary, with this conventional type of input device, there is a decoupling of human physical-movement with control of one's viewpoint in the synthetic-world.
The aforementioned limitations of the described prior art have been addressed by the developing art of head-tracked head-mounted displays (HTHMDs). Early embodiments of such HTHMDs were developed for flight simulators. A. M. Spooner teaches such apparatus and methods for training pilots in U.S. Pat. Nos. 4,315,240; 4,31.5,241; 4,340,878; 4,347,508 and 4,349,815. Such art in combination with high-speed computing means make the simulated effect of flying an airplane so compelling that pilots can effectively be trained without leaving the ground. Other examples of HTHMDs include those available from Virtual Research, Sega of America, Inc., and others, as described in the June 1993 issue of Popular Science, at pages 83-86 and 124-125. The HTHMD art convincingly connects the natural movement of the human in the physical world to his (her) viewpoint in the virtual-world. So that, for example, when a user physically turns around, she sees what is behind her in the virtual-world. There is strong correspondence between her expected change in virtual-view based on her physical-movement sensory feedback and the actual virtual-view experienced.
Notwithstanding these benefits, systems employing such head-tracked helmet-style devices do present a variety of problems with their practical use.
Since the human head varies greatly in size and shape, it is difficult to fit a standard-sized head-mounted display to the general population. An unobtrusive fit is important since the experience of virtual reality is greatly enhanced when the user is unencumbered by the system. In addition, their small size requires them to be fitted with small, expensive, and typically heavy, high-resolution visual displays. In multiple-user environments, HTHMD are not hygienic. Respiratory contagions as well as hair parasites such as lice and scabies can be transmitted via these devices. In short, head-mounted displays tend to be heavy, poorly balanced on the head, non-hygienic and slow to setup.
Furthermore, the current art of head-mounted displays requires an information-conducting means in the form of an umbilical-like cable attachment to the computing means which is prone to failure and inherently awkward to manipulate. Also, the intuitive interface for navigating in the virtual-world provided by the HTHMD only works within the range of the head-tracking device. Activity is, in effect, confined to an area relatively proximate to the head-tracker reference sensor means. If the virtual-world is the world of an airplane cockpit, this is less of a problem than if the virtual-world is a cityscape. Common forms of positional tracker attachments to HTHMD are available from Polhemus Corp. and Ascension Technology Corp. Virtual-navigation or positional change outside of the relatively small envelope of the above devices requires some other, non-physically intuitive means of movement in the virtual world. One such device, disclosed in U.S. Pat. No. 4,988,981 (referenced above), comprises a glove with integrated position sensing means for finger movements and a second external position sensing means for hand rotation and translation in three dimensions. The image displayed within the virtual-environment is that of a crude, working hand. The user may point in a direction and then activate a means of "flying" in that virtual-direction. The user may also manipulate objects within the virtual-environment by seemingly grasping them. The operation of the glove device is only natural and intuitive with the movement of the fingers and the axial rotation of the hand. Human locomotion is still accomplished by virtual-flying or gliding directed by a hand which has no physical-world analog.
Lastly, HTHMD systems often pose problems with the use of and amount of physical space needed to contain the volume in which the user will physically move. For example, if the intended use is for public entertainment arcades, then the relatively large kiosk footprint leads to poor profitability because the revenue per area ratio is reduced. Or if the intended user has limited work space such as a scientist or engineer limited to his work cubicle, then HTHMDs are less feasible. However, in the case of a flight simulator, the HTHMD works well because the pilot trainee is already physically constrained to movement within the relatively small cockpit.
Consider, however, the problem of moving about in a virtual-building, for example to go from room to room inspecting furniture. It would be appropriate to design the virtual reality system to have a physical volume matching the size of the virtual-building. In practice, usually either some method of changing the scale of physical-movement to virtual-movement or the use of some alternative mode of movement representation is employed to deal with this problem. For example, by making hand gestures with the aforementioned instrumented glove, virtual-buttons might be pressed to virtually-fly around the building. Then, in order to inspect the furniture closely, one may need to change the interface mode back to the original one whereby human physical-motion is mapped one to one with virtual-motion. Thus to navigate large virtual spaces among the rooms or pieces of furniture in the rooms, conventional technology would use one of the aforementioned devices and methods of virtual-movement which do not inherently exhibit congruence of physical-motion with virtual-motion. This mode changing in order to accommodate gross and fine movement is a significant drawback to the practical implementation of virtual reality systems.
In sum, even though the head-mounted displays of the current art do effectively pair motion in the physical world with that of the virtual world, they present a variety of problems with their use.
In daily life when we want to see something, we just move our bodies around by walking, bending, turning and so forth until we can see it. It is by such naturally coordinated movement that we change our present viewpoint to some new, desired viewpoint. The terms "viewpoint," "view," and "area-of-interest" as used herein will be defined with reference to FIGS. 10, 11 and 12. FIG. 10 is a schematic diagram of a section of railroad track as seen from the top. At point A, the user's "viewpoint" (the position and orientation from where the user is looking) is depicted by the arrow emanating from the circle around A. FIG. 11 depicts the "view" or visual content from viewpoint A, that is, what is actually seen from viewpoint A. In order for the viewpoint to move from A to B, the user must move along the track to point B and then look left or, in other words, change her position to B and orient left. FIG. 12 depicts an approximate view from B. Another way to state the above scenario is to say that the movement of the user's viewpoint from A to B is the result of having changed her "area-of-interest" from that depicted in FIG. 11 to that of FIG. 12. While the above terms refer specifically to vision, other senses such as audition or olfaction, which involve similar spatial-perceptual problems for the human in virtual-words are meant to be within their scope.
While we have very sophisticated and naturally integrated human hardware for dealing with the physical-world, we have no such natural interface to support behavior in simulated or virtual-worlds. The prior art in the form of head-mounted displays, tactile feedback devices, instrumented gloves and body suits and so forth, are improvements to the art of more closely connecting the human to a non-physical world. They provide proprioceptic and other state-of-the-body position and orientation information to the user as useful feedback for control of virtual movement. Unfortunately, the prior art still suffers from the deficiencies a ready outlined. The present invention offers a number of advantages including improved intuitive and effective feedback control in a small physical space.